A noise audit of human-labeled benchmarks for machine commonsense reasoning.

With the advent of large language models, evaluating and benchmarking these systems on important AI problems has taken on newfound importance. Such benchmarking typically involves comparing the predictions of a system against human labels (or a single 'ground-truth'). However, much recent work in psychology has suggested that most tasks involving significant human judgment can have non-trivial degrees of noise. In his book, Kahneman suggests that noise may be a much more significant component of inaccuracy compared to bias, which has been studied more extensively in the AI community. This article proposes a detailed noise audit of human-labeled benchmarks in machine commonsense reasoning, an important current area of AI research. We conduct noise audits under two important experimental conditions: one in a smaller-scale but higher-quality labeling setting, and another in a larger-scale, more realistic online crowdsourced setting. Using Kahneman's framework of noise, our results consistently show non-trivial amounts of level, pattern, and system noise, even in the higher-quality setting, with comparable results in the crowdsourced setting. We find that noise can significantly influence the performance estimates that we obtain of commonsense reasoning systems, even if the 'system' is a human; in some cases, by almost 10 percent. Labeling noise also affects performance estimates of systems like ChatGPT by more than 4 percent. Our results suggest that the default practice in the AI community of assuming and using a 'single' ground-truth, even on problems requiring seemingly straightforward human judgment, may warrant empirical and methodological re-visiting.

Demographic and socioeconomic determinants of access to care: A subgroup disparity analysis using new equity-focused measurements.

Disparities in healthcare access and utilization associated with demographic and socioeconomic status hinder advancement of health equity. Thus, we designed a novel equity-focused approach to quantify variations of healthcare access/utilization from the expectation in national target populations. We additionally applied survey-weighted logistic regression models, to identify factors associated with usage of a particular type of health care. To facilitate generation of analysis datasets, we built an National Health and Nutrition Examination Survey (NHANES) knowledge graph to help automate source-level dynamic analyses across different survey years and subjects' characteristics. We performed a cross-sectional subgroup disparity analysis of 2013-2018 NHANES on U.S. adults for receipt of diabetes treatments and vaccines against Hepatitis A (HAV), Hepatitis B (HBV), and Human Papilloma (HPV). Results show that in populations with hemoglobin A1c level ≥6%, patients with non-private insurance were less likely to receive newer and more beneficial antidiabetic medications; being Asian further exacerbated these disparities. For widely used drugs such as insulin, Asians experienced insignificant disparities in odds of prescription compared to White patients but received highly inadequate treatments with regard to their distribution in U.S. diabetic population. Vaccination rates were associated with some demographic/socioeconomic factors but not the others at different degrees for different diseases. For instance, while equity scores increase with rising education levels for HBV, they decrease with rising wealth levels for HPV. Among women vaccinated against HPV, minorities and poor communities usually received Cervarix while non-Hispanic White and higher-income groups received the more comprehensive Gardasil vaccine. Our study identified and quantified the impact of determinants of healthcare utilization for antidiabetic medications and vaccinations. Our new methods for semantics-aware disparity analysis of NHANES data could be readily generalized to other public health goals to support more rapid identification of disparities and development of policies, thus advancing health equity.

TG-CSR: A human-labeled dataset grounded in nine formal commonsense categories.

Machine Common Sense Reasoning is the subfield of Artificial Intelligence that aims to enable machines to behave or make decisions similarly to humans in everyday and ordinary situations. To measure progress, benchmarks in the form of question-answering datasets have been developed and published in the community to evaluate machine commonsense models, including large language models. We describe the individual label data produced by six human annotators originally used in computing ground truth for the Theoretically-Grounded Commonsense Reasoning (TG-CSR) benchmark's composing datasets. According to a set of instructions, annotators were provided with spreadsheets containing the original TG-CSR prompts and asked to insert labels in specific spreadsheet cells during annotation sessions. TG-CSR data is organized in JSON files, individual raw label data in a spreadsheet file, and individual normalized label data in JSONL files. The release of individual labels can enable the analysis of the labeling process itself, including studies of noise and consistency across annotators.

Semantically enabling clinical decision support recommendations.

BACKGROUND: Clinical decision support systems have been widely deployed to guide healthcare decisions on patient diagnosis, treatment choices, and patient management through evidence-based recommendations. These recommendations are typically derived from clinical practice guidelines created by clinical specialties or healthcare organizations. Although there have been many different technical approaches to encoding guideline recommendations into decision support systems, much of the previous work has not focused on enabling system generated recommendations through the formalization of changes in a guideline, the provenance of a recommendation, and applicability of the evidence. Prior work indicates that healthcare providers may not find that guideline-derived recommendations always meet their needs for reasons such as lack of relevance, transparency, time pressure, and applicability to their clinical practice. RESULTS: We introduce several semantic techniques that model diseases based on clinical practice guidelines, provenance of the guidelines, and the study cohorts they are based on to enhance the capabilities of clinical decision support systems. We have explored ways to enable clinical decision support systems with semantic technologies that can represent and link to details in related items from the scientific literature and quickly adapt to changing information from the guidelines, identifying gaps, and supporting personalized explanations. Previous semantics-driven clinical decision systems have limited support in all these aspects, and we present the ontologies and semantic web based software tools in three distinct areas that are unified using a standard set of ontologies and a custom-built knowledge graph framework: (i) guideline modeling to characterize diseases, (ii) guideline provenance to attach evidence to treatment decisions from authoritative sources, and (iii) study cohort modeling to identify relevant research publications for complicated patients. CONCLUSIONS: We have enhanced existing, evidence-based knowledge by developing ontologies and software that enables clinicians to conveniently access updates to and provenance of guidelines, as well as gather additional information from research studies applicable to their patients' unique circumstances. Our software solutions leverage many well-used existing biomedical ontologies and build upon decades of knowledge representation and reasoning work, leading to explainable results.

Informing clinical assessment by contextualizing post-hoc explanations of risk prediction models in type-2 diabetes.

Medical experts may use Artificial Intelligence (AI) systems with greater trust if these are supported by 'contextual explanations' that let the practitioner connect system inferences to their context of use. However, their importance in improving model usage and understanding has not been extensively studied. Hence, we consider a comorbidity risk prediction scenario and focus on contexts regarding the patients' clinical state, AI predictions about their risk of complications, and algorithmic explanations supporting the predictions. We explore how relevant information for such dimensions can be extracted from Medical guidelines to answer typical questions from clinical practitioners. We identify this as a question answering (QA) task and employ several state-of-the-art Large Language Models (LLM) to present contexts around risk prediction model inferences and evaluate their acceptability. Finally, we study the benefits of contextual explanations by building an end-to-end AI pipeline including data cohorting, AI risk modeling, post-hoc model explanations, and prototyped a visual dashboard to present the combined insights from different context dimensions and data sources, while predicting and identifying the drivers of risk of Chronic Kidney Disease (CKD) - a common type-2 diabetes (T2DM) comorbidity. All of these steps were performed in deep engagement with medical experts, including a final evaluation of the dashboard results by an expert medical panel. We show that LLMs, in particular BERT and SciBERT, can be readily deployed to extract some relevant explanations to support clinical usage. To understand the value-add of the contextual explanations, the expert panel evaluated these regarding actionable insights in the relevant clinical setting. Overall, our paper is one of the first end-to-end analyses identifying the feasibility and benefits of contextual explanations in a real-world clinical use case. Our findings can help improve clinicians' usage of AI models.

Differences in Cancer Phenotypes Among Frequent CHEK2 Variants and Implications for Clinical Care-Checking CHEK2.

Importance: Germline CHEK2 pathogenic variants (PVs) are frequently detected by multigene cancer panel testing (MGPT), but our understanding of PVs beyond c.1100del has been limited. Objective: To compare cancer phenotypes of frequent CHEK2 PVs individually and collectively by variant type. Design, Setting, and Participants: This retrospective cohort study was carried out in a single diagnostic testing laboratory from 2012 to 2019. Overall, 3783 participants with CHEK2 PVs identified via MGPT were included. Medical histories of cancer in participants with frequent PVs, negative MGPT (wild type), loss-of-function (LOF), and missense were compared. Main Outcomes and Measures: Participants were stratified by CHEK2 PV type. Descriptive statistics were summarized including median (IQR) for continuous variables and proportions for categorical characteristics. Differences in age and proportions were assessed with Wilcoxon rank sum and Fisher exact tests, respectively. Frequencies, odds ratios (ORs), 95% confidence intervals were calculated, and P values were corrected for multiple comparisons where appropriate. Results: Of the 3783 participants with CHEK2 PVs, 3473 (92%) were female and most reported White race. Breast cancer was less frequent in participants with p.I157T (OR, 0.66; 95% CI, 0.56-0.78; P<.001), p.S428F (OR, 0.59; 95% CI. 0.46-0.76; P<.001), and p.T476M (OR, 0.74; 95% CI, 0.56-0.98; P = .04) PVs compared with other PVs and an association with nonbreast cancers was not found. Following the exclusion of p.I157T, p.S428F, and p.T476M, participants with monoallelic CHEK2 PV had a younger age at first cancer diagnosis (P < .001) and were more likely to have breast (OR, 1.83; 95% CI, 1.66-2.02; P < .001), thyroid (OR, 1.63; 95% CI, 1.26-2.08; P < .001), and kidney cancer (OR, 2.57; 95% CI, 1.75-3.68; P < .001) than the wild-type cohort. Participants with a CHEK2 PV were less likely to have a diagnosis of colorectal cancer (OR, 0.62; 95% CI, 0.51-0.76; P < .001) compared with those in the wild-type cohort. There were no significant differences between frequent CHEK2 PVs and c.1100del and no differences between CHEK2 missense and LOF PVs. Conclusions and Relevance: CHEK2 PVs, with few exceptions (p.I157T, p.S428F, and p.T476M), were associated with similar cancer phenotypes irrespective of variant type. CHEK2 PVs were not associated with colorectal cancer, but were associated with breast, kidney, and thyroid cancers. Compared with other CHEK2 PVs, the frequent p.I157T, p.S428F, and p.T476M alleles have an attenuated association with breast cancer and were not associated with nonbreast cancers. These data may inform the genetic counseling and care of individuals with CHEK2 PVs.

FAIR and Interactive Data Graphics from a Scientific Knowledge Graph.

Graph databases capture richly linked domain knowledge by integrating heterogeneous data and metadata into a unified representation. Here, we present the use of bespoke, interactive data graphics (bar charts, scatter plots, etc.) for visual exploration of a knowledge graph. By modeling a chart as a set of metadata that describes semantic context (SPARQL query) separately from visual context (Vega-Lite specification), we leverage the high-level, declarative nature of the SPARQL and Vega-Lite grammars to concisely specify web-based, interactive data graphics synchronized to a knowledge graph. Resources with dereferenceable URIs (uniform resource identifiers) can employ the hyperlink encoding channel or image marks in Vega-Lite to amplify the information content of a given data graphic, and published charts populate a browsable gallery of the database. We discuss design considerations that arise in relation to portability, persistence, and performance. Altogether, this pairing of SPARQL and Vega-Lite-demonstrated here in the domain of polymer nanocomposite materials science-offers an extensible approach to FAIR (findable, accessible, interoperable, reusable) scientific data visualization within a knowledge graph framework.

Transforming the study of organisms: Phenomic data models and knowledge bases.

The rapidly decreasing cost of gene sequencing has resulted in a deluge of genomic data from across the tree of life; however, outside a few model organism databases, genomic data are limited in their scientific impact because they are not accompanied by computable phenomic data. The majority of phenomic data are contained in countless small, heterogeneous phenotypic data sets that are very difficult or impossible to integrate at scale because of variable formats, lack of digitization, and linguistic problems. One powerful solution is to represent phenotypic data using data models with precise, computable semantics, but adoption of semantic standards for representing phenotypic data has been slow, especially in biodiversity and ecology. Some phenotypic and trait data are available in a semantic language from knowledge bases, but these are often not interoperable. In this review, we will compare and contrast existing ontology and data models, focusing on nonhuman phenotypes and traits. We discuss barriers to integration of phenotypic data and make recommendations for developing an operationally useful, semantically interoperable phenotypic data ecosystem.

The Semantic Data Dictionary - An Approach for Describing and Annotating Data.

It is common practice for data providers to include text descriptions for each column when publishing datasets in the form of data dictionaries. While these documents are useful in helping an end-user properly interpret the meaning of a column in a dataset, existing data dictionaries typically are not machine-readable and do not follow a common specification standard. We introduce the Semantic Data Dictionary, a specification that formalizes the assignment of a semantic representation of data, enabling standardization and harmonization across diverse datasets. In this paper, we present our Semantic Data Dictionary work in the context of our work with biomedical data; however, the approach can and has been used in a wide range of domains. The rendition of data in this form helps promote improved discovery, interoperability, reuse, traceability, and reproducibility. We present the associated research and describe how the Semantic Data Dictionary can help address existing limitations in the related literature. We discuss our approach, present an example by annotating portions of the publicly available National Health and Nutrition Examination Survey dataset, present modeling challenges, and describe the use of this approach in sponsored research, including our work on a large NIH-funded exposure and health data portal and in the RPI-IBM collaborative Health Empowerment by Analytics, Learning, and Semantics project. We evaluate this work in comparison with traditional data dictionaries, mapping languages, and data integration tools.

Polymer Nanocomposite Data: Curation, Frameworks, Access, and Potential for Discovery and Design.

With the advent of the materials genome initiative (MGI) in the United States and a similar focus on materials data around the world, a number of materials data resources and associated vocabularies, tools, and repositories have been developed. While the majority of systems focus on slices of computational data with an emphasis on metallic alloys, NanoMine is an open source platform with the goal of curating and storing widely varying experimental data on polymer nanocomposites (polymers doped with nanoparticles) and providing access to characterization and analysis tools with the long-term objective of promoting facile nanocomposite design. Data on over 2500 samples from the literature and individual laboratories has been curated to date into NanoMine, including 230 samples from the papers bound in this virtual issue. This virtual issue represents an experiment of the flexibility of the data repository to capture the unique experimental metadata requirements of many data sets at one time and to challenge the authors to participate in the curation of their research data associated with a given publication. In principle, NanoMine offers a FAIR platform in which data published in papers becomes directly Findable and Accessible via simple search tools, with open metadata standards that are Interoperable with larger materials data registries, and allows easy Reuse of data, e.g. benchmarking against new results. Our hope is that with time, platforms such as this one could capture much of the newly published data on materials and form nodes in an interconnected materials data ecosystem which would allow researchers to robustly archive their data, add to the growing body of readily accessible data, and enable new forms of discovery by application of data analysis and design tools.

Identifying Ingredient Substitutions Using a Knowledge Graph of Food.

People can affect change in their eating patterns by substituting ingredients in recipes. Such substitutions may be motivated by specific goals, like modifying the intake of a specific nutrient or avoiding a particular category of ingredients. Determining how to modify a recipe can be difficult because people need to 1) identify which ingredients can act as valid replacements for the original and 2) figure out whether the substitution is "good" for their particular context, which may consider factors such as allergies, nutritional contents of individual ingredients, and other dietary restrictions. We propose an approach to leverage both explicit semantic information about ingredients, encapsulated in a knowledge graph of food, and implicit semantics, captured through word embeddings, to develop a substitutability heuristic to rank plausible substitute options automatically. Our proposed system also helps determine which ingredient substitution options are "healthy" using nutritional information and food classification constraints. We evaluate our substitutability heuristic, diet-improvement ingredient substitutability heuristic (DIISH), using a dataset of ground-truth substitutions scraped from ingredient substitution guides and user reviews of recipes, demonstrating that our approach can help reduce the human effort required to make recipes more suitable for specific dietary needs.

Knowledge Extraction of Cohort Characteristics in Research Publications.

When healthcare providers review the results of a clinical trial study to understand its applicability to their practice, they typically analyze how well the characteristics of the study cohort correspond to those of the patients they see. We have previously created a study cohort ontology to standardize this information and make it accessible for knowledge-based decision support. The extraction of this information from research publications is challenging, however, given the wide variance in reporting cohort characteristics in a tabular representation. To address this issue, we have developed an ontology-enabled knowledge extraction pipeline for automatically constructing knowledge graphs from the cohort characteristics found in PDF-formatted research papers. We evaluated our approach using a training and test set of 41 research publications and found an overall accuracy of 83.3% in correctly assembling the knowledge graphs. Our research provides a promising approach for extracting knowledge more broadly from tabular information in research publications.

Clustering of co-occurring conditions in autism spectrum disorder during early childhood: A retrospective analysis of medical claims data.

Individuals with autism spectrum disorder (ASD) are frequently affected by co-occurring medical conditions (COCs), which vary in severity, age of onset, and pathophysiological characteristics. The presence of COCs contributes to significant heterogeneity in the clinical presentation of ASD between individuals and a better understanding of COCs may offer greater insight into the etiology of ASD in specific subgroups while also providing guidance for diagnostic and treatment protocols. This study retrospectively analyzed medical claims data from a private United States health plan between years 2000 and 2015 to investigate patterns of COC diagnoses in a cohort of 3,278 children with ASD throughout their first 5 years of enrollment compared to 279,693 children from the general population without ASD diagnoses (POP cohort). Three subgroups of children with ASD were identified by k-means clustering using these COC patterns. The first cluster was characterized by generally high rates of COC diagnosis and comprised 23.7% (n = 776) of the cohort. Diagnoses of developmental delays were dominant in the second cluster containing 26.5% (n = 870) of the cohort. Children in the third cluster, making up 49.8% (n = 1,632) of the cohort, had the lowest rates of COC diagnosis, which were slightly higher than rates observed in the POP cohort. A secondary analysis using these data found that gastrointestinal and immune disorders showed similar longitudinal patterns of prevalence, as did seizure and sleep disorders. These findings may help to better inform the development of diagnostic workup and treatment protocols for COCs in children with ASD. Autism Res 2019, 12: 1272-1285. © 2019 International Society for Autism Research, Wiley Periodicals, Inc. LAY SUMMARY: Medical conditions that co-occur with autism spectrum disorder (ASD) vary significantly from person to person. This study analyzed patterns in diagnosis of co-occurring conditions from medical claims data and observed three subtypes of children with ASD. These results may aid with screening for co-occurring conditions in children with ASD and with understanding ASD subtypes.

Identifying Windows of Susceptibility by Temporal Gene Analysis.

Increased understanding of developmental disorders of the brain has shown that genetic mutations, environmental toxins and biological insults typically act during developmental windows of susceptibility. Identifying these vulnerable periods is a necessary and vital step for safeguarding women and their fetuses against disease causing agents during pregnancy and for developing timely interventions and treatments for neurodevelopmental disorders. We analyzed developmental time-course gene expression data derived from human pluripotent stem cells, with disease association, pathway, and protein interaction databases to identify windows of disease susceptibility during development and the time periods for productive interventions. The results are displayed as interactive Susceptibility Windows Ontological Transcriptome (SWOT) Clocks illustrating disease susceptibility over developmental time. Using this method, we determine the likely windows of susceptibility for multiple neurological disorders using known disease associated genes and genes derived from RNA-sequencing studies including autism spectrum disorder, schizophrenia, and Zika virus induced microcephaly. SWOT clocks provide a valuable tool for integrating data from multiple databases in a developmental context with data generated from next-generation sequencing to help identify windows of susceptibility.

Gastrointestinal Symptoms and Oral Antibiotic Use in Children with Autism Spectrum Disorder: Retrospective Analysis of a Privately Insured U.S. Population.

A retrospective analysis of administrative claims data from a large U.S. health insurer was performed to study a potential association between oral antibiotic use during early childhood and occurrence of later gastrointestinal (GI) symptoms in children with autism spectrum disorder (ASD). Among 3253 children with ASD, 37.0% had a GI-related diagnosis during the last 2 years of their 5-year health coverage enrollment period, compared to 20.0% of 278,370 children from the general population without an ASD diagnosis. Greater numbers of oral antibiotic fills during the first 3 years of enrollment were found to significantly increase the hazard rate of having a later GI-related diagnosis (adjusted hazard ratio 1.48; 95% confidence interval 1.34, 1.63) in children both with and without ASD.

Investigating Plasma Amino Acids for Differentiating Individuals with Autism Spectrum Disorder and Typically Developing Peers.

BACKGROUND: Plasma amino acid measurements have been extensively investigated in individuals with autism spectrum disorder (ASD). Results thus far have been inconclusive as studies generally disagree on which amino acids are different in individuals with ASD versus their typically developing (TD) peers, due in part to methodological limitations of several studies. METHOD: This paper investigates plasma amino acids in children and adults with ASD using data from Arizona State University's Comprehensive Nutritional and Dietary Intervention Study. Measurements from 64 individuals with ASD and 49 TD controls were analyzed using univariate and multivariate statistical techniques. RESULTS: Univariate analysis indicated increased median levels of glutamate (+21%, p=0.014) and serine (+8%, p=0.043), and increased mean levels of hydroxyproline (+17%, p=0.018) for the ASD cohort, although these differences were insignificant after correcting for multiple comparisons. A multivariate approach was used to classify study participants into ASD/TD cohorts using Fisher discriminant analysis (FDA) and its nonlinear extension, kernel Fisher discriminant analysis (KFDA). Model fitting with FDA using all available measurements produced Type I and Type II errors of 27.0% and 27.8%, respectively. KFDA was most effective when using hydroxyproline, leucine, and threonine as inputs; however, leave-one-out cross-validation with this nonlinear model only resulted in 70.3% sensitivity and 77.6% specificity. CONCLUSIONS: The finding of elevated glutamate in ASD is in agreement with several other studies. Overall, however, these results suggest that plasma amino acid measurements are of limited use for purposes of ASD classification, which may explain some of the inconsistencies in results presented in the literature.

Big and disparate data: considerations for pediatric consortia.

PURPOSE OF REVIEW: Increasingly, there is a need for examining exposure disease associations in large, diverse datasets to understand the complex determinants of pediatric disease and disability. Recognizing that children's health research consortia will be important sources of big data, it is crucial for the pediatric research community to be knowledgeable about the challenges and opportunities that they will face. The present review will provide examples of existing children's health consortia, highlight recent pooled analyses conducted by children's health research consortia, address common challenges of pooled analyses, and provide recommendations to advance collective research efforts in pediatric research. RECENT FINDINGS: Formal consortia and other collective-science initiatives are increasingly being created to share individual data from a set of relevant epidemiological studies to address a common research topic under the concept that the joint effort of many individual groups can accomplish far more than working alone. There are practical challenges to the participation of investigators within consortia that need to be addressed in order for them to work. SUMMARY: Researchers who access consortia with data centers will be able to go far beyond their initial hypotheses and potentially accomplish research that was previously thought infeasible or too costly.

On the Use of Multivariate Methods for Analysis of Data from Biological Networks.

Data analysis used for biomedical research, particularly analysis involving metabolic or signaling pathways, is often based upon univariate statistical analysis. One common approach is to compute means and standard deviations individually for each variable or to determine where each variable falls between upper and lower bounds. Additionally, p-values are often computed to determine if there are differences between data taken from two groups. However, these approaches ignore that the collected data are often correlated in some form, which may be due to these measurements describing quantities that are connected by biological networks. Multivariate analysis approaches are more appropriate in these scenarios, as they can detect differences in datasets that the traditional univariate approaches may miss. This work presents three case studies that involve data from clinical studies of autism spectrum disorder that illustrate the need for and demonstrate the potential impact of multivariate analysis.

Finding melanoma drugs through a probabilistic knowledge graph.

Metastatic cutaneous melanoma is an aggressive skin cancer with some progression-slowing treatments but no known cure. The omics data explosion has created many possible drug candidates; however, filtering criteria remain challenging, and systems biology approaches have become fragmented with many disconnected databases. Using drug, protein and disease interactions, we built an evidence-weighted knowledge graph of integrated interactions. Our knowledge graph-based system, ReDrugS, can be used via an application programming interface or web interface, and has generated 25 high-quality melanoma drug candidates. We show that probabilistic analysis of systems biology graphs increases drug candidate quality compared to non-probabilistic methods. Four of the 25 candidates are novel therapies, three of which have been tested with other cancers. All other candidates have current or completed clinical trials, or have been studied in in vivo or in vitro. This approach can be used to identify candidate therapies for use in research or personalized medicine.

A framework for evaluating and utilizing medical terminology mappings.

Use of medical terminologies and mappings across them are considered to be crucial pre-requisites for achieving interoperable eHealth applications. Built upon the outcomes of several research projects, we introduce a framework for evaluating and utilizing terminology mappings that offers a platform for i) performing various mappings strategies, ii) representing terminology mappings together with their provenance information, and iii) enabling terminology reasoning for inferring both new and erroneous mappings. We present the results of the introduced framework from SALUS project where we evaluated the quality of both existing and inferred terminology mappings among standard terminologies.